Skip to contentSkip to conversation

image_types_sparse: use ext2simg_android for ext* sparse images

PATCH In review AshishKumar Mishra 2 participants 6 messages Jul 21 – Jul 23

lore
Share & tools
lore
#0 lore raw

v3 cover image_types_sparse: use ext2simg_android for ext* sparse images

This series switches ext* sparse image generation from the generic img2simg to ext2simg_android from e2fsprogs contrib/android. Unlike img2simg,which scans the raw image for zero-filled blocks, ext2simg_android understands the ext* filesystem structure and uses the block bitmap to determine sparse areas, which is faster and produces smaller sparse images.
This reduces flashing time for Android RFS/fastboot images and cuts cost in CI/CD pipelines. img2simg is retained for non-ext types such as f2fs.

1. e2fsprogs-ext4sparse-native: a standalone native recipe that compiles ext2simg.c against android-tools' libsparse.
2. android-tools: export libsparse/libbase/liblog headers and libs to the sysroot (under ${includedir}/sparse) so the recipe above can link against them.
3. image_types_sparse.bbclass: branch CONVERSION_CMD:sparse by filesystem type, calling ext2simg_android for ext* and falling back to img2simg otherwise.

Changes in v3:
- Android tools has been moved away from selinux dynamic-layers
- Reworked into a standalone e2fsprogs-ext4sparse recipe in meta-oe as per feedback on the RFC
- Included the previously-missing e2fsprogs-ext4sparse recipe file that was absent from the v2 (RFC_V2) attachment.

Changes in v2 (RFC_V2):
- Split the single cross-layer PoC into a dedicated ext2simg_android recipe rather than patching e2fsprogs-native directly.

RFC (v1):
- Initial cross-layer proof of concept using a manual ${CC} call in do_compile:append and LD_LIBRARY_PATH for runtime linking.

AshishKumar Mishra (3):
e2fsprogs-ext4sparse-native: compiles ext2simg.c using android-tools-native
android-tools: export libsparse/libbase/liblog headers and libs to sysroot
image_types_sparse: switch ext* conversion to ext2simg_android

 meta-oe/classes/image_types_sparse.bbclass    | 16 ++++--
 .../android-tools/android-tools_%.bbappend    | 54 +++++++++++++++++++
 .../e2fsprogs-ext4sparse-native_1.47.4.bb     | 34 ++++++++++++
 3 files changed, 101 insertions(+), 3 deletions(-)
 create mode 100644 meta-oe/recipes-devtools/android-tools/android-tools_%.bbappend
 create mode 100644 meta-oe/recipes-devtools/e2fsprogs-ext4sparse/e2fsprogs-ext4sparse-native_1.47.4.bb
Signature · 1 line
2.43.0
#1 lore raw

PATCH v3 1/3 e2fsprogs-ext4sparse-native: compiles ext2simg.c using android-tools-native

Add a custom do_compile step to build the ext2simg_android utility (ext2simg) when building for the native class.
This can be used to generate ext* based sparse image

---
 .../e2fsprogs-ext4sparse-native_1.47.4.bb     | 32 +++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100644 meta-oe/recipes-devtools/e2fsprogs-ext4sparse/e2fsprogs-ext4sparse-native_1.47.4.bb
meta-oe/recipes-devtools/e2fsprogs-ext4sparse/e2fsprogs-ext4sparse-native_1.47.4.bb new file +32 -0
new file mode 100644
index 0000000000..04f9aaaf2b
@@ -0,0 +1,32 @@
+SUMMARY = "Native ext sparse conversion utility from e2fsprogs"
+DESCRIPTION = "Builds only ext2simg for host-side ext* sparse image conversion"
+HOMEPAGE = "http://e2fsprogs.sourceforge.net/"
+LICENSE = "Apache-2.0"
+LIC_FILES_CHKSUM = "file://contrib/android/ext2simg.c;beginline=1;endline=14;md5=fc8a826484cc82548adeaad2579adfca"
+
+DEPENDS = "e2fsprogs-native android-tools-native"
+
+SRC_URI = "git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git;branch=master;protocol=https"
+SRCREV = "ece89fac4603e400155b7bbf6326284f8511bca9"
+
+inherit native
+
+do_compile() {
+    bbnote "Compiling ext2simg.c with native toolchain"
+    # Source directory for ext2simg.c in the e2fsprogs tree
+    SRC_EXT2SIMG="${S}/contrib/android"
+
+    INCLUDES="-I${S}/lib -I${B}/lib -I${SRC_EXT2SIMG}/lib"
+
+    # STAGING_LIBDIR_NATIVE: where android-tools-native installed libsparse
+    LIBS="-L${B}/lib -L${RECIPE_SYSROOT_NATIVE}${libdir} -L${RECIPE_SYSROOT_NATIVE}${libdir}/android -lsparse -lext2fs -lz -lcom_err"
+
+    ${CC} ${CFLAGS} ${LDFLAGS} -o "${B}/ext2simg_android" "${SRC_EXT2SIMG}/ext2simg.c" ${INCLUDES} ${LIBS}
+    bbnote "ext2simg_android compilation finished"
+}
+
+do_install() {
+    install -d ${D}${bindir}
+    install -m 0755 ${B}/ext2simg_android ${D}${bindir}/ext2simg_android
+}
+
Signature · 1 line
2.43.0
#2 lore raw

PATCH v3 2/3 android-tools: export libsparse/libbase/liblog headers and libs to sysroot

In reply to AshishKumar Mishra

Export libsparse, libbase, and liblog headers and libraries to the sysroot. Currently, these are often internal to the build.
Reorganise header installation to ${includedir}/sparse to match the expected include paths for e2fsprogs contrib tools.

---
 .../android-tools/android-tools_%.bbappend    | 54 +++++++++++++++++++
 1 file changed, 54 insertions(+)
 create mode 100644 meta-oe/recipes-devtools/android-tools/android-tools_%.bbappend
meta-oe/recipes-devtools/android-tools/android-tools_%.bbappend new file +54 -0
new file mode 100644
index 0000000000..50c54145b8
@@ -0,0 +1,54 @@
+FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
+
+# e2fsprogs expecting headers in sparse/ subdirectory
+do_install:append:class-native() {
+    # to resolve the "Multiple shlib providers" conflict.
+    rm -f ${D}${libdir}/android/libsparse.so*
+    rm -f ${D}${libdir}/android/libbase.so*
+    rm -f ${D}${libdir}/android/liblog.so*
+
+    if [ -d "${S}/system/core/libsparse/include/sparse" ]; then
+        install -d ${D}${includedir}/sparse
+        cp -r ${S}/system/core/libsparse/include/sparse/* ${D}${includedir}/sparse/
+    else
+        bbfatal "Sparse headers not found in ${S}/system/core/libsparse/include/sparse"
+    fi
+
+    install -d ${D}${libdir}
+    for lib in libsparse libbase liblog; do
+        if [ -f "${S}/debian/out/system/${lib}.so" ]; then
+            bbnote "Installing ${lib} from ${S}/debian/out/system/${lib}.so"
+            install -m 0755 ${S}/debian/out/system/${lib}.so ${D}${libdir}/${lib}.so.0
+            ln -sf ${lib}.so.0 ${D}${libdir}/${lib}.so
+        elif [ -f "${S}/debian/out/system/${lib}.so.0" ]; then
+            bbnote "Installing ${lib} from ${S}/debian/out/system/${lib}.so.0"
+            install -m 0755 ${S}/debian/out/system/${lib}.so.0 ${D}${libdir}/${lib}.so.0
+            ln -sf ${lib}.so.0 ${D}${libdir}/${lib}.so
+        else
+            bberror "${lib} not found in ${S}/debian/out/system"
+        fi
+    done
+
+    install -d ${D}${libdir}/android
+    for lib in libsparse libbase liblog; do
+        if [ -f "${D}${libdir}/${lib}.so.0" ]; then
+            ln -sf ../${lib}.so.0 ${D}${libdir}/android/${lib}.so.0
+            ln -sf ../${lib}.so ${D}${libdir}/android/${lib}.so
+        fi
+    done
+}
+
+FILES:${PN}-dev += " \
+    ${includedir}/sparse \
+    ${libdir}/lib*.so \
+    ${libdir}/android/lib*.so \
+"
+
+FILES:${PN} += " \
+    ${libdir}/lib*.so.* \
+    ${libdir}/android/lib*.so.* \
+"
+
+SYSROOT_DIRS:append:class-native = " ${includedir} ${libdir}"
+
+DEPENDS:append:class-native = " googletest-native"
Signature · 1 line
2.43.0
#3 lore raw

PATCH v3 3/3 image_types_sparse: switch ext* conversion to ext2simg_android

In reply to AshishKumar Mishra

We have RFC discussion at
https://lists.openembedded.org/g/openembedded-devel/message/124499
https://lists.openembedded.org/g/openembedded-devel/message/125497

Move ext* sparse image generation from 'img2simg' to ' ext2simg_android'

---
 meta-oe/classes/image_types_sparse.bbclass | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)
meta-oe/classes/image_types_sparse.bbclass +13 -3
index 5416c2a019..5c1d92bd18 100644
@@ -13,11 +13,21 @@ DELETE_RAWIMAGE_AFTER_SPARSE_CMD ??= "0"
 
 CONVERSION_CMD:sparse = " \
     truncate --no-create --size=%${SPARSE_BLOCK_SIZE} "${IMAGE_NAME}.${type}"; \
-    img2simg -s "${IMAGE_NAME}.${type}" "${IMAGE_NAME}.${type}.sparse" ${SPARSE_BLOCK_SIZE}; \
+    case '${type}' in \
+        ext*) \
+            bbnote 'Running e2fsprogs-derived ext2simg_android..' ; \
+            ext2simg_android '${IMAGE_NAME}.${type}' '${IMAGE_NAME}.${type}.simg' || bberror 'ext2simg_android failed' \
+            ;; \
+        *) \
+            bbnote 'Generating sparse image for non-ext filesystem...'; \
+            img2simg -s '${IMAGE_NAME}.${type}' '${IMAGE_NAME}.${type}.sparse' ${SPARSE_BLOCK_SIZE}; \
+            ;; \
+    esac; \
     if [ "${DELETE_RAWIMAGE_AFTER_SPARSE_CMD}" = "1" ]; then \
         rm -f ${IMAGE_NAME}.${type};\
-        bbwarn "Raw file ${IMAGE_NAME}.${type} removed" ;\
+        bbnote "Raw file ${IMAGE_NAME}.${type} removed" ;\
     fi;\
  "
 
-CONVERSION_DEPENDS_sparse = "android-tools-native"
+CONVERSION_DEPENDS_sparse = "android-tools-native e2fsprogs-ext4sparse-native"
+do_image_ext4[depends] += "e2fsprogs-ext4sparse-native:do_populate_sysroot"
Signature · 1 line
2.43.0
#5 lore raw

Hi Team ,
Can team please take this in consideration in master branch
If team have any specific feedback , i can quickly work and submit accordingly

Thanks ,
Ashish